Data Filtering and Transformation

**Referenced Files in This Document** - [.eleventy.js](file://.eleventy.js) - [package.json](file://package.json) - [README.md](file://README.md) - [src/_data/site.json](file://src/_data/site.json) - [src/_data/homepage.json](file://src/_data/homepage.json) - [src/_data/polling.js](file://src/_data/polling.js) - [src/content/news/news.11tydata.json](file://src/content/news/news.11tydata.json) - [src/content/team/team.11tydata.json](file://src/content/team/team.11tydata.json) - [src/content/cases/cases.11tydata.json](file://src/content/cases/cases.11tydata.json)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document explains Eleventy’s data processing capabilities in this project, focusing on built-in and custom filters, transformation functions, and how they integrate with collections and data files. It covers filter implementation patterns, parameter handling, return value expectations, filter chaining, and performance considerations. Practical examples demonstrate complex data transformations, conditional filtering, and data sanitization. Debugging techniques and optimization strategies for large datasets are included.

Project Structure

Eleventy configuration and data live primarily under the repository root and the src directory:

  • Eleventy configuration defines passthrough copies, watches, shortcodes, filters, collections, and transforms.
  • Data is loaded from src/_data and content front matter/data files via per-content 11tydata files.
  • Templates consume filters and collections to render content.
graph TB
A[".eleventy.js<br/>Filters, Collections, Transforms"] --> B["src/_data/*.json<br/>Static data"]
A --> C["src/_data/polling.js<br/>Async data"]
A --> D["src/content/**/*.11tydata.*<br/>Per-content data"]
E["Templates (.njk, .md)"] --> F["Collections"]
E --> G["Filters"]
F --> H["Rendered Pages"]
G --> H
B --> E
C --> E
D --> E

Diagram sources

  • [.eleventy.js:1-283](file://.eleventy.js#L1-L283)

Section sources

  • [.eleventy.js:1-283](file://.eleventy.js#L1-L283)

Core Components

This project exposes a set of Eleventy filters and collections that enable robust data transformation and rendering. Filters are registered in the Eleventy configuration and applied within templates to transform arrays, strings, dates, and content.

Key filters implemented in this project:

  • limit: Returns the first n items of an array.
  • skip: Skips the first n items of an array.
  • dateFormat/date: Formats dates with multiple presets.
  • slug: Produces URL-safe slugs from strings.
  • truncate: Truncates text with a configurable suffix.
  • nl2br: Converts newline characters to HTML line breaks.
  • firstParagraph: Extracts the first paragraph from HTML content.
  • dump/log: Debug helpers for inspecting data.
  • split: Splits strings by a separator.
  • readingTime: Estimates reading time based on word count.
  • sortBy: Sorts arrays by a property with ascending/descending support.
  • where: Filters arrays by property equality.
  • unique: Deduplicates arrays by a property value.

Collections:

  • news, cases, newsletters, teamMembers, services, knowledge: Content collections filtered and sorted by glob patterns and metadata.

Section sources

  • [.eleventy.js:46-164](file://.eleventy.js#L46-L164)
  • [.eleventy.js:170-210](file://.eleventy.js#L170-L210)
  • [README.md:591-626](file://README.md#L591-L626)

Architecture Overview

The data pipeline integrates Eleventy’s configuration, data loading, and template rendering:

  • Data sources include static JSON files, JavaScript async data, and per-content 11tydata files.
  • Collections assemble content items and apply sorting and filtering.
  • Filters transform data inline during rendering.
  • Transforms post-process rendered HTML.
sequenceDiagram
participant Cfg as ".eleventy.js"
participant Data as "Data Files"
participant Coll as "Collections"
participant Tpl as "Templates"
participant Out as "Output"
Cfg->>Data : Load _data and 11tydata
Cfg->>Coll : Register collections
Tpl->>Coll : Query collections
Tpl->>Cfg : Apply filters
Cfg-->>Tpl : Transformed data
Tpl->>Out : Rendered HTML

Diagram sources

  • [.eleventy.js:1-283](file://.eleventy.js#L1-L283)

Detailed Component Analysis

Built-in Filters: Implementation Patterns and Behavior

Each filter follows a consistent pattern:

  • Parameter validation: Guards against non-array inputs or missing values.
  • Safe defaults: Returns empty arrays or empty strings when inputs are invalid.
  • Deterministic output: Ensures consistent return types for template consumption.
  • Optional parameters: Uses defaults for optional arguments (e.g., truncate suffix, sortBy order).

Examples of filter categories and behaviors:

  • Array slicing: limit, skip
  • String transformation: slug, truncate, nl2br, split
  • Content extraction: firstParagraph
  • Date formatting: dateFormat/date
  • Reading time estimation: readingTime
  • Sorting and filtering: sortBy, where, unique
  • Debugging: dump, log

Return value expectations:

  • limit/skip: Array
  • slug/truncate/nl2br/split: String
  • firstParagraph: String
  • dateFormat/date: String
  • readingTime: String
  • sortBy/where/unique: Array
  • dump: String (JSON)
  • log: Original value (with side-effect logging)

Parameter handling:

  • Many filters accept optional parameters with sensible defaults.
  • Filters validate inputs early and return safe defaults to avoid runtime errors.

Filter chaining:

  • Filters can be chained in templates to compose transformations (e.g., applying slug after truncating and before joining).
  • Order matters: for example, sorting before limiting often yields different results than limiting before sorting.

Practical examples (described):

  • Limit and skip: Use limit to cap featured items and skip to paginate.
  • Sort and filter: sortBy by date or category, then where to isolate items by a flag.
  • Unique: Deduplicate by category or author prior to rendering lists.
  • Reading time: Combine readingTime with truncate to summarize content previews.
  • Date formatting: Use dateFormat with presets to present human-readable dates consistently.

Section sources

  • [.eleventy.js:46-164](file://.eleventy.js#L46-L164)
  • [README.md:591-608](file://README.md#L591-L608)

Custom Filter Creation

To add a new filter:

  • Define a function that accepts the appropriate parameters and returns a transformed value.
  • Register it via eleventyConfig.addFilter in the Eleventy configuration file.
  • Apply it in templates using the pipe syntax.

Implementation tips:

  • Validate inputs early and return safe defaults.
  • Keep filters pure when possible to simplify testing and caching.
  • Consider performance for large datasets (see Performance Considerations).

Filter chaining patterns:

  • Chain multiple filters to build complex transformations incrementally.
  • Prefer ordering that minimizes intermediate allocations (e.g., filter before sort).

Section sources

  • [.eleventy.js:46-164](file://.eleventy.js#L46-L164)

Collections and Data Integration

Collections assemble content items and apply sorting and filtering:

  • news: Globbed MD files sorted by date descending.
  • cases: Globbed MD files.
  • newsletters: Globbed MD files sorted by date descending.
  • teamMembers: Globbed MD files sorted by an order field.
  • services: Globbed MD files sorted by an order field.
  • knowledge: Globbed MD files sorted by date descending.

Per-content 11tydata files:

  • news.11tydata.json disables permalinks for news content.
  • team.11tydata.json disables permalinks for team content.
  • cases.11tydata.json disables permalinks for cases content.

Data files:

  • site.json: Site-wide metadata and social links.
  • homepage.json: Homepage-specific labels and CTAs.
  • polling.js: Async polling data with fallback support.

These data sources feed into templates and filters to produce rendered pages.

Section sources

  • [.eleventy.js:170-210](file://.eleventy.js#L170-L210)
  • [src/content/news/news.11tydata.json:1-2](file://src/content/news/news.11tydata.json#L1-L2)
  • [src/content/team/team.11tydata.json:1-2](file://src/content/team/team.11tydata.json#L1-L2)
  • [src/content/cases/cases.11tydata.json:1-2](file://src/content/cases/cases.11tydata.json#L1-L2)
  • [src/_data/site.json:1-20](file://src/_data/site.json#L1-L20)
  • [src/_data/homepage.json:1-23](file://src/_data/homepage.json#L1-L23)
  • [src/_data/polling.js:1-17](file://src/_data/polling.js#L1-L17)

Data Flow and Rendering

flowchart TD
Start(["Template Renders"]) --> GetData["Load Data<br/>_data + 11tydata"]
GetData --> GetColl["Access Collections"]
GetColl --> ApplyFilters["Apply Filters<br/>limit/sortBy/where/unique/etc."]
ApplyFilters --> TransformHTML["Optional Transforms<br/>Obsidian syntax, HTML minify"]
TransformHTML --> Output(["Final HTML"])

Diagram sources

  • [.eleventy.js:170-210](file://.eleventy.js#L170-L210)
  • [.eleventy.js:217-239](file://.eleventy.js#L217-L239)
  • [.eleventy.js:242-261](file://.eleventy.js#L242-L261)

Dependency Analysis

  • Eleventy core dependency is declared in package.json.
  • Production HTML minification depends on html-minifier-terser.
  • Collections depend on content glob patterns and front matter/order fields.
  • Filters depend on input types and optional parameters.
graph LR
Pkg["package.json"] --> E11ty["@11ty/eleventy"]
Pkg --> Min["html-minifier-terser"]
Cfg[".eleventy.js"] --> Filters["Custom Filters"]
Cfg --> Collections["Content Collections"]
Filters --> Tpl["Templates"]
Collections --> Tpl
Tpl --> Out["Output"]

Diagram sources

  • [package.json:14-20](file://package.json#L14-L20)
  • [.eleventy.js:242-261](file://.eleventy.js#L242-L261)

Section sources

  • [package.json:14-20](file://package.json#L14-L20)
  • [.eleventy.js:242-261](file://.eleventy.js#L242-L261)

Performance Considerations

  • Prefer filtering before sorting when possible to reduce comparison work.
  • Use limit early to cap result sets for expensive operations.
  • Avoid repeated heavy computations inside loops; cache computed values when feasible.
  • For large datasets, consider pagination via skip and limit to reduce rendering load.
  • Use unique judiciously; deduplication adds O(n) overhead per property.
  • The readingTime filter estimates based on word counts; keep content clean to avoid miscounts.
  • In production, rely on the HTML minify transform to reduce payload sizes.

[No sources needed since this section provides general guidance]

Troubleshooting Guide

Common issues and resolutions:

  • Non-array inputs to limit/skip/sortBy/where/unique: These filters return safe defaults (empty arrays) to prevent errors. Verify upstream data shapes.
  • Empty or undefined strings to slug/truncate/nl2br/firstParagraph/split: These return safe defaults (empty strings). Confirm data availability.
  • Invalid dates to dateFormat/date: Returns empty strings for invalid inputs. Validate date fields in front matter.
  • dump/log: dump produces JSON strings for debugging; log prints values to the console and returns the original value. Use sparingly in templates to avoid noisy logs.
  • Reading time anomalies: Ensure content is clean of HTML tags when relying on word counts; consider stripping tags before applying readingTime.
  • Chaining filters: If output is unexpected, reorder filters to align with intended transformations.

Section sources

  • [.eleventy.js:46-164](file://.eleventy.js#L46-L164)

Conclusion

This project demonstrates a practical, extensible approach to Eleventy data processing using custom filters and collections. By validating inputs, returning safe defaults, and leveraging filter chaining, templates remain resilient and efficient. For large datasets, combine pagination, early limiting, and judicious use of unique to maintain performance while delivering rich, dynamic content.

[No sources needed since this section summarizes without analyzing specific files]

Appendices

Filter Reference and Usage Notes

  • limit: array | limit(n)
  • skip: array | skip(n)
  • dateFormat/date: date | dateFormat('iso'|'short'|'year'|'long')
  • slug: string | slug
  • truncate: string | truncate(length, suffix)
  • nl2br: string | nl2br
  • firstParagraph: content | firstParagraph
  • dump: object | dump
  • log: value | log
  • split: string | split(separator)
  • readingTime: content | readingTime
  • sortBy: array | sortBy(property, 'asc'|'desc')
  • where: array | where(property, value)
  • unique: array | unique(property)

Section sources

  • [README.md:591-608](file://README.md#L591-L608)